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Abstract 



With the growing interest in comparing student achievement 
state-by-state, there is a need to develop better techaniques for 
making such comparisons. Approaches available now, particularly 
the use of SAT and ACT scores, have obvious shortcomings but non- 
e-the-less continue to be prominent because there is nothing more 
defensible. In this paper, the role of the federal government in 
encouraging new approaches is discussed, particularly the explo- 
ration of the use of state assessments in cross-state compari- 
sons, and the extension of KAEP to a nationally representative 
sample for this purpose. 
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State-by-state comparisons appear inescapable, no matter the 
cautions voiced about the interpetations to be made. The Reno 
Gazette- Journal of March 3, 1986 published under the headline 
■Nevada at bottom of education rankings* a recounting of the 
latest findings from the Secretary's wall chart, with a comment 
from the Chief State School Officer that the rankings are 
■probably not a good indicator of (Nevada's) overall. . .school 
program" but should not be dismissed. On the other hand, the 
Milwaukee Journal reported ecstatic comments about Wisconsin 
coming in first in the ACT rankings. 

While competition and comparisons have long been a part of our 
culture, state-by-state comparisons of educational achievement 
have only recently been made. The Council of Chief State School 
Officers traditionally opposed such comparisons for student 
achievement, and The National Assessment of Educational Progress 
was designed as it was to make such comparisons impossible. 

The purpose of this paper is to review the role of the federal 
government, specifically the Department of Education, in the 
current discussion of cross-state comparisons. Also, I will 
attempt to give some indication of the directions the 
Department of Education might take in the near future. However, I 
must include two caveats. First, the Department is now in the 
process of formulating a long-term policy regarding state 
comparisons, which will be part of a much larger strategy for 
d*ta collection. Very little cnn be said for certain at this 
time. Second, I am not directly involved in the current 
discussions. Since the reorganization of OERI relocated all of 
the regular data collection programs under the Center for 
Statistics, the Office of Research where I am located has had no 
direct connection with these activities. While this may serve to 
make me a more objective reporter, it certainly does not help to 
make me a better informed one. 
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For the critical achievement outcome measures, however, the 
indicators project had some of the same difficulties as the wall 
Start! Although it drew on both the NAEP data and the 
international comparison studies, the Indicators proaect was 
lolSed to See Iat and ACT scores for state comparisons. Thus some 
of the most important policy implications of the project depended 
on these badly flawed indicators. 

One way to solve the problem of achievement indicators would 
be to expand the current NAEP testing program to draw a 
representative sample of students in each state. The contractor 
ets and its advisory board, are quite willing to do this* 
brt'the cost would be much higher than the current contract. 
Also! the intermittent schedule of testing that NAEP uses makes 
it less desireable as an indicator. 

Recognizing these difficulties, and the fact that a 
substantial amount of achievement testing is being fjne by state 
and local school districts, NCES commissioned the Center for the 
Study of Evaluation to do a study on the feasibility of using 
state data as national indicators of educational quality. 
Analyzing whatever documentation could be collected from tne 
Tails) whether actual tests, test specif icjtions, or other 
information, the CSE study group concluded that there 
be enough common content among state tests to make comparisons 
possible. It would be feasible to develop a common core of items, 
scaled by IRT methods, that could serve as a linking mechanism. 
However, this could only be done through an evolutionary process, 
as the differences in procedures across states would make it 
"possible for even most of the states to immediately participate 
in such an effort, no matter how conceived. 

This brings us to the Fall of 1985, which coincided with the 
reorganization of OERI and the consolidation of all statistical 
data collection under the Center for Educational statistics. 
Three other events, should also be noted. 

o Secretary Bennett and Assistant Secretary Finn called for a 
substantial increase in the NAEP testing program-more fluent 
testing, particularly in the basic skills; testing of adults; and 
increased subject matter testing. 

o The Council for Chief State School Officers reversed its 
position against state-by-state achievement comparisons. In order 
to devise means for implementing this change, CCSSO established a 
Center on Assessment and had a committee studying the ways that 
cross-state achievement comparisons might be made. 

o Three southern states completed a mini-NAEP study, wh ,ere a 
shortened version of the NAEP reading assessment was a c m i nis ^ rea 
by these states, at their expense, as a feasibility study. This 
is to be repeated in 1986 with a NAEP writing assessment. 
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Current Plans 

Consolidation of the statistical data collections has 
permitted CES to look comprehensively at what that total data set 
is, and how it might be altered to meet needs more effectively. 
CES has been exploring this since early 1985, recently publishing 
a Working Paper on a Plan for the Redesign of the Elementary and 
Secondary Data Collection Program(March 27, 1986). A number of 
papers had been commisssioned to analyze the existing data set. 
Authors of these papers concluded that the data available from 
CES are deficient in the following respects: 

o Comprehensiveness: there are notable gaps in cove "9e» 
particularly for student achievement, teacher data, and school 
finance. With respect to NAEP, the gaps particularly noted were 
the infrequency of testing for most academic sub3ects and the 
lack of state representative samples. CE S also collects 
achievement information in the National Educational Longitudinal 
Study (NELS) and the High School and Beyond study but achievement 
is not a central component of either and the content coverage is 
quite limited. 

o Integration- The data collection efforts are generally 
conducted independently, making it impossible to establish 
interrelationships. 

o Inaccuracy and Noncomparability-Errors are notable in the data 
set due, in part, to the various agencies involved in data 
collection with differing definitions and procedures. 

o Representativeness- much of the data can not be disaggregated 
below a national level. This is particularly true in regard to 
achievement and other student characteristics. 

o Timeliness-There have been significant delays in collect ng, 
compiling and summarizing data, often leading to the publication 
of separate indicators from several different years in the same 
reports. 

In response to these problems, CES has proposed a redesign of 
the Elementary/Secondary Integrated Data System. This would 
include a sample of 4200 public and private schools with 
stratification at the 4th, 8th and 12th grades. The school would 
become the critical sampling unit 'th samples of students, 
teachers, parents and administrate drawn from these schools. 
About 25% of the sample schools wo. .»i be replaced annually. 

In regard to achievement testing, a revised NAEP program would 
be established, including shortened versions of the tests. 
Reading and mathematics would be alternated yearly, with a second 
academic subject assessed each year. These instruments would 
yield a single score in the particular subject matter. The intent 
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<«= *~ woon assessment time in each area to an hour or less, but 
rtiU bS aM. "^continue tb. tin. trend analysis from previous 



assessments. 



A major strength of the new design is its greater capacity for 

assrawsa: ijss^fffi^^^ 

be special concentration on the target grades. 

would b! responsible for collecting the school level data. 

Pinallv. the redesign calls for a good deal of background 
information on students, teachers, schools and prog rams. This 
rnntinues a trend that was started some years ago with NAEP, but 
represents a departure from the previous NCES practices. 

However, there are also a number of questions and concerns 
that will need to be addressed. 

1 Ts it possible to create a shortened or streamlined NAEP test 
for ?he trill delired? In order to relate back to Previous 
fdministrations of the NAEP instruments, the tests will need to 
IT equated, most°likely through IRT methods 8 £°£ en | d "*to n be 
fcesfc has alreadlv been successfully scaled, but tnis is yet co 
done in aainea.tics. Whether it is possible in other areas, 
particularly in science, is unknown. 

2 What happens to the current NAEP assessment? There is an 
implication to be drawn that NAEP as it is now done would be 
phased out , but this is not certain. It appears clear that item 
development would be continued for the new instruments, 
reaardlels of what happens to the current NAEP. It could be that 
I version of NAeI similar to what is now available would be 
continued for states that do not wish to build their own 
assessment program. How much financial responsibility for this 
the federal govermaent would assume is not Known. 

3. Can state comparisons be made? At the outset, there will be 
no Plans to make such comparisons in student achievement. State 
SftmniJisons would be possible for other indicators that are not 
saSed It tSe school P le?el. Initial samples would be drawn in 
Sn P s?atet, bit the schools would not be f tat %"P""2 n ative ' 
except perhaps in a few of the larger states. Achievement 
comparisons would eventually come through an evolutionary 
Access, hopefully with considerable assistance from the states. 
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If states are to become involved in the data collection, then 
there will need to be some new procedures developed. The 
trade-off of quality versus cost will need to be explored. It 
might be possible for the federal government to do -ome of the 
data collection for its purposes, and then provide Lhe 
instruments to the states for their purposes. 

4. How useful will the new achievement test be to the states? 
There is certainly a strong hope that the states will play a 
major role in this effort; the tests are to be designed so they 
could readily be incorporated into the states* assessment 
programs. States should be able to scale their instruments to the 
national test and thus give more fine-grained breakdowns within 
their state while providing the data for comparisons across 
states on the national standard. However, this will be possible 
only for the abbreviated tests. Just how much enthusiasm, and 
resources, the states will have for comparisons based on the 
streamlined measures is open to question. 

I draw two conclusions from the recent federal 
proposal. First, there appears to be a shift in the naef 
Assessment Program, making it a federal rather than a national 
assessment. Revisons suggested in NAEP will neet federal needs 
more adequately than is now the case. Wheth r these revisions 
will meet others" needs as adequately is not as clear. 

Second, it does appear clear that the states will have to take 
a more active role in making cross-state comparisons that are 
meaningful for them. That this is already occurring is an 
indication that the federal government may not need to be a 
central force in actually creating such comparisons. 
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